Video coding

ABSTRACT

Improved video coding is described to encode video data within a sequence of video frames. To this end, at least a portion of a reference frame is encoded to include motion information associated with the portion of the reference frame. At least a portion of a predictable frame that includes video data predictively correlated to said portion of said reference frame is defined based on the motion information. At least said portion of the predictable frame is encoded without including corresponding motion information and including mode identifying data. The mode identifying data indicate that the encoded portion of the predictable frame can be directly derived using at least the motion information associated with the portion of the reference frame.

RELATED PATENT APPLICATIONS

This patent application is a continuation of U.S. patent applicationSer. No. 10/186,284 (“parent”), titled “Video Coding Methods andApparatuses”, filed on Jun. 27, 2002, now U.S. Pat. No. 7,003,035, andhereby incorporated by reference. The parent application Ser. No.10/186,284 claimed the benefit of: (1) U.S. Provisional PatentApplication No. 60/376,005, filed Apr. 26, 2002; and (2) U.S.Provisional Patent Application No. 60/352,127, filed Jan. 25, 2002.

BACKGROUND

The motivation for increased coding efficiency in video coding has ledto the adoption in the Joint Video Team (JVT) (a standards body) of morerefined and complicated models and modes describing motion informationfor a given macroblock. These models and modes tend to make betteradvantage of the temporal redundancies that may exist within a videosequence. See, for example, ITU-T, Video Coding Expert Group (VCEG),“JVT Coding—(ITU-T H.26L & ISO/IEC JTC1 Standard) —Working Draft Number2 (WD-2)”, ITU-T JVT-B 118, March 2002; and/or Heiko Schwarz and ThomasWiegand, “Tree-structured macroblock partition”, Doc. VCEG-N17, December2001.

The recent models include, for example, multi-frame indexing of themotion vectors, increased sub-pixel accuracy, multi-referencing, andtree structured macroblock and motion assignment, according to whichdifferent sub areas of a macroblock are assigned to different motioninformation. Unfortunately these models tend to also significantlyincrease the required percentage of bits for the encoding of motioninformation within sequence. Thus, in some cases the models tend toreduce the efficacy of such coding methods.

Even though, in some cases, motion vectors are differentially encodedversus a spatial predictor, or even skipped in the case of zero motionwhile having no residue image to transmit, this does not appear to besufficient for improved efficiency.

It would, therefore, be advantageous to further reduce the bits requiredfor the encoding of motion information, and thus of the entire sequence,while at the same time not significantly affecting quality.

Another problem that is also introduced by the adoption of such modelsand modes is that of determining the best mode among all possiblechoices, for example, given a goal bitrate, encoding/quantizationparameters, etc. Currently, this problem can be partially solved by theuse of cost measures/penalties depending on the mode and/or thequantization to be used, or even by employing Rate DistortionOptimization techniques with the goal of minimizing a Lagrangianfunction.

Such problems and others become even more significant, however, in thecase of Bidirectionally Predictive (B) frames where a macroblock may bepredicted from both future and past frames. This essentially means thatan even larger percentage of bits may be required for the encoding ofmotion vectors.

Hence, there is a need for improved method and apparatuses for use incoding (e.g., encoding and/or decoding) video data.

SUMMARY

Video coding methods and apparatuses are provided that make use ofvarious models and/or modes to significantly improve coding efficiencyespecially for high/complex motion sequences. The methods andapparatuses take advantage of the temporal and/or spatial correlationsthat may exist within portions of the frames, e.g., at the Macroblockslevel, etc. For example, motion vectors for a macroblock may betemporally and/or spatially predicted from previously encoded motioninformation. The methods and apparatuses tend to significantly reducethe amount of data required for encoding motion information whileretaining or even improving video image quality.

This Summary is provided to introduce a selection of concepts in asimplified form that are further described below in the detaileddescription. This Summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used as an aid in determining the scope of the claimed subjectmatter.

In view of the above, improved video coding is described to encode videodata within a sequence of video frames. To this end, at least a portionof a reference frame is encoded to include motion information associatedwith the portion of the reference frame. At least a portion of apredictable frame that includes video data predictively correlated tosaid portion of said reference frame is defined based on the motioninformation. At least said portion of the predictable frame is encodedwithout including corresponding motion information and including modeidentifying data. The mode identifying data indicate that the encodedportion of the predictable frame can be directly derived using at leastthe motion information associated with the portion of the referenceframe

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention is illustrated by way of example and notlimitation in the figures of the accompanying drawings. The same numbersare used throughout the figures to reference like components and/orfeatures.

FIG. 1 is a block diagram depicting an exemplary computing environmentthat is suitable for use with certain implementations of the presentinvention.

FIG. 2 is a block diagram depicting an exemplary representative devicethat is suitable for use with certain implementations of the presentinvention.

FIG. 3 is an illustrative diagram depicting a Direct Motion Projectiontechnique suitable for use in B Frame coding, in accordance with certainexemplary implementations of the present invention.

FIG. 4 is an illustrative diagram depicting a Direct P and B codingtechniques within a sequence of video frames, in accordance with certainexemplary implementations of the present invention.

FIG. 5 is an illustrative diagram depicting Direct Motion Prediction forcollocated macroblocks having identical motion information, inaccordance with certain exemplary implementations of the presentinvention.

FIG. 6 is an illustrative diagram depicting motion projection, inaccordance with certain exemplary implementations of the presentinvention.

FIG. 7 is an illustrative diagram depicting the usage of accelerationinformation in Direct Motion Projection, in accordance with certainexemplary implementations of the present invention.

FIG. 8 is an illustrative diagram depicting a Direct Pixel Projectiontechnique suitable for use in B Frame coding, in accordance with certainexemplary implementations of the present invention.

FIG. 9 is an illustrative diagram depicting a Direct Pixel Projectiontechnique suitable for use in P Frame coding, in accordance with certainexemplary implementations of the present invention.

FIG. 10 is a block diagram depicting an exemplary conventional videoencoder.

FIG. 11 is a block diagram depicting an exemplary conventional videodecoder.

FIG. 12 is a block diagram depicting an exemplary improved video encoderusing Direct Prediction, in accordance with certain exemplaryimplementations of the present invention.

FIG. 13 is a block diagram depicting an exemplary improved video decoderusing Direct Prediction, in accordance with certain exemplaryimplementations of the present invention.

FIG. 14 is an illustrative diagram depicting a Direct Pixel/BlockProjection technique, in accordance with certain exemplaryimplementations of the present invention.

FIG. 15 is an illustrative diagram depicting a Direct Motion Projectiontechnique suitable for use in B Frame coding, in accordance with certainexemplary implementations of the present invention.

FIG. 16 is an illustrative diagram depicting motion vector predictions,in accordance with certain exemplary implementations of the presentinvention.

FIG. 17 is an illustrative diagram depicting interlace coding techniquesfor P frames, in accordance with certain exemplary implementations ofthe present invention.

FIG. 18 is an illustrative diagram depicting interlace coding techniquesfor B frames, in accordance with certain exemplary implementations ofthe present invention.

FIG. 19 is an illustrative diagram depicting a scheme for coding jointfield/frame images, in accordance with certain exemplary implementationsof the present invention.

DETAILED DESCRIPTION

In accordance with certain aspects of the present invention, methods andapparatuses are provided for coding (e.g., encoding and/or decoding)video data. The methods and apparatuses can be configured to enhance thecoding efficiency of “interlace” or progressive video coding streamingtechnologies. In certain implementations, for example, with regard tothe current H.26L standard, so called “P-frames” have been significantlyenhanced by introducing several additional macroblock Modes. In somecases it may now be necessary to transmit up to 16 motion vectors permacroblock. Certain aspects of the present invention provide a way ofencoding these motion vectors. For example, as described below, Direct Pprediction techniques can be used to select the motion vectors ofcollocated pixels in the previous frame.

While these and other exemplary methods and apparatuses are described,it should be kept in mind that the techniques of the present inventionare not limited to the examples described and shown in the accompanyingdrawings, but are also clearly adaptable to other similar existing andfuture video coding schemes, etc.

Before introducing such exemplary methods and apparatuses, anintroduction is provided in the following section for suitable exemplaryoperating environments, for example, in the form of a computing deviceand other types of devices/appliances.

Exemplary Operational Environments:

Turning to the drawings, wherein like reference numerals refer to likeelements, the invention is illustrated as being implemented in asuitable computing environment. Although not required, the inventionwill be described in the general context of computer-executableinstructions, such as program modules, being executed by a personalcomputer.

Generally, program modules include routines, programs, objects,components, data structures, etc. that perform particular tasks orimplement particular abstract data types. Those skilled in the art willappreciate that the invention may be practiced with other computersystem configurations, including hand-held devices, multi-processorsystems, microprocessor based or programmable consumer electronics,network PCs, minicomputers, mainframe computers, portable communicationdevices, and the like.

The invention may also be practiced in distributed computingenvironments where tasks are performed by remote processing devices thatare linked through a communications network. In a distributed computingenvironment, program modules may be located in both local and remotememory storage devices.

FIG. 1 illustrates an example of a suitable computing environment 120 onwhich the subsequently described systems, apparatuses and methods may beimplemented. Exemplary computing environment 120 is only one example ofa suitable computing environment and is not intended to suggest anylimitation as to the scope of use or functionality of the improvedmethods and systems described herein. Neither should computingenvironment 120 be interpreted as having any dependency or requirementrelating to any one or combination of components illustrated incomputing environment 120.

The improved methods and systems herein are operational with numerousother general purpose or special purpose computing system environmentsor configurations. Examples of well known computing systems,environments, and/or configurations that may be suitable include, butare not limited to, personal computers, server computers, thin clients,thick clients, hand-held or laptop devices, multiprocessor systems,microprocessor-based systems, set top boxes, programmable consumerelectronics, network PCs, minicomputers, mainframe computers,distributed computing environments that include any of the above systemsor devices, and the like.

As shown in FIG. 1, computing environment 120 includes a general-purposecomputing device in the form of a computer 130. The components ofcomputer 130 may include one or more processors or processing units 132,a system memory 134, and a bus 136 that couples various systemcomponents including system memory 134 to processor 132.

Bus 136 represents one or more of any of several types of busstructures, including a memory bus or memory controller, a peripheralbus, an accelerated graphics port, and a processor or local bus usingany of a variety of bus architectures. By way of example, and notlimitation, such architectures include Industry Standard Architecture(ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA)bus, Video Electronics Standards Association (VESA) local bus, andPeripheral Component Interconnects (PCI) bus also known as Mezzaninebus.

Computer 130 typically includes a variety of computer readable media.Such media may be any available media that is accessible by computer130, and it includes both volatile and non-volatile media, removable andnon-removable media.

In FIG. 1, system memory 134 includes computer readable media in theform of volatile memory, such as random access memory (RAM) 140, and/ornon-volatile memory, such as read only memory (ROM) 138. A basicinput/output system (BIOS) 142, containing the basic routines that helpto transfer information between elements within computer 130, such asduring start-up, is stored in ROM 138. RAM 140 typically contains dataand/or program modules that are immediately accessible to and/orpresently being operated on by processor 132.

Computer 130 may farther include other removable/non-removable,volatile/non-volatile computer storage media. For example, FIG. 1illustrates a hard disk drive 144 for reading from and writing to anon-removable, non-volatile magnetic media (not shown and typicallycalled a “hard drive”), a magnetic disk drive 146 for reading from andwriting to a removable, non-volatile magnetic disk 148 (e.g., a “floppydisk”), and an optical disk drive 150 for reading from or writing to aremovable, non-volatile optical disk 152 such as a CD-ROM/R/RW,DVD-ROM/R/RW/+R/RAM or other optical media. Hard disk drive 144,magnetic disk drive 146 and optical disk drive 150 are each connected tobus 136 by one or more interfaces 154.

The drives and associated computer-readable media provide nonvolatilestorage of computer readable instructions, data structures, programmodules, and other data for computer 130. Although the exemplaryenvironment described herein employs a hard disk, a removable magneticdisk 148 and a removable optical disk 152, it should be appreciated bythose skilled in the art that other types of computer readable mediawhich can store data that is accessible by a computer, such as magneticcassettes, flash memory cards, digital video disks, random accessmemories (RAMs), read only memories (ROM), and the like, may also beused in the exemplary operating environment.

A number of program modules may be stored on the hard disk, magneticdisk 148, optical disk 152, ROM 138, or RAM 140, including, e.g., anoperating system 158, one or more application programs 160, otherprogram modules 162, and program data 164.

The improved methods and systems described herein may be implementedwithin operating system 158, one or more application programs 160, otherprogram modules 162, and/or program data 164.

A user may provide commands and information into computer 130 throughinput devices such as keyboard 166 and pointing device 168 (such as a“mouse”). Other input devices (not shown) may include a microphone,joystick, game pad, satellite dish, serial port, scanner, camera, etc.These and other input devices are connected to the processing unit 132through a user input interface 170 that is coupled to bus 136, but maybe connected by other interface and bus structures, such as a parallelport, game port, or a universal serial bus (USB).

A monitor 172 or other type of display device is also connected to bus136 via an interface, such as a video adapter 174. In addition tomonitor 172, personal computers typically include other peripheraloutput devices (not shown), such as speakers and printers, which may beconnected through output peripheral interface 175.

Computer 130 may operate in a networked environment using logicalconnections to one or more remote computers, such as a remote computer182. Remote computer 182 may include many or all of the elements andfeatures described herein relative to computer 130.

Logical connections shown in FIG. 1 are a local area network (LAN) 177and a general wide area network (WAN) 179. Such networking environmentsare commonplace in offices, enterprise-wide computer networks,intranets, and the Internet.

When used in a LAN networking environment, computer 130 is connected toLAN 177 via network interface or adapter 186. When used in a WANnetworking environment, the computer typically includes a modem 178 orother means for establishing communications over WAN 179. Modem 178,which may be internal or external, may be connected to system bus 136via the user input interface 170 or other appropriate mechanism.

Depicted in FIG. 1, is a specific implementation of a WAN via theInternet. Here, computer 130 employs modem 178 to establishcommunications with at least one remote computer 182 via the Internet180.

In a networked environment, program modules depicted relative tocomputer 130, or portions thereof, may be stored in a remote memorystorage device. Thus, e.g., as depicted in FIG. 1, remote applicationprograms 189 may reside on a memory device of remote computer 182. Itwill be appreciated that the network connections shown and described areexemplary and other means of establishing a communications link betweenthe computers may be used.

Attention is now drawn to FIG. 2, which is a block diagram depictinganother exemplary device 200 that is also capable of benefiting from themethods and apparatuses disclosed herein. Device 200 is representativeof any one or more devices or appliances that are operatively configuredto process video and/or any related types of data in accordance with allor part of the methods and apparatuses described herein and theirequivalents. Thus, device 200 may take the form of a computing device asin FIG. 1, or some other form, such as, for example, a wireless device,a portable communication device, a personal digital assistant, a videoplayer, a television, a DVD player, a CD player, a karaoke machine, akiosk, a digital video projector, a flat panel video display mechanism,a set-top box, a video game machine, etc. In this example, device 200includes logic 202 configured to process video data, a video data source204 configured to provide vide data to logic 202, and at least onedisplay module 206 capable of displaying at least a portion of the videodata for a user to view. Logic 202 is representative of hardware,firmware, software and/or any combination thereof. In certainimplementations, for example, logic 202 includes acompressor/decompressor (codec), or the like. Video data source 204 isrepresentative of any mechanism that can provide, communicate, output,and/or at least momentarily store video data suitable for processing bylogic 202. Video reproduction source is illustratively shown as beingwithin and/or without device 200. Display module 206 is representativeof any mechanism that a user might view directly or indirectly and seethe visual results of video data presented thereon. Additionally, incertain implementations, device 200 may also include some form orcapability for reproducing or otherwise handling audio data associatedwith the video data. Thus, an audio reproduction module 208 is shown.

With the examples of FIGS. 1 and 2 in mind, and others like them, thenext sections focus on certain exemplary methods and apparatuses thatmay be at least partially practiced using with such environments andwith such devices.

Direct Prediction for Predictive (P) and Bidirectionally Predictive (B)Frames in Video Coding:

This section presents a new highly efficient Inter Macroblock type thatcan significantly improve coding efficiency especially for high/complexmotion sequences. This Inter Macroblock new type takes advantage of thetemporal and spatial correlations that may exist within frames at themacroblock level, and as a result can significantly reduce the bitsrequired for encoding motion information while retaining or evenimproving quality.

Direct Prediction

The above mentioned problems and/or others are at least partially solvedherein by the introduction of a “Direct Prediction Mode” wherein,instead of encoding the actual motion information, both forward and/orbackward motion vectors are derived directly from the motion vectorsused in the correlated macroblock of the subsequent reference frame.

This is illustrated, for example, in FIG. 3, which shows three videoframes, namely a P frame 300, a B frame 302 and P frame 304,corresponding to times t, t+1, and t+2, respectively, Also illustratedin FIG. 3 are macroblocks within frames 300, 302 and 304 and exemplarymotion vector (MV) information. Here, the frames have x and ycoordinates associated with them. The motion vector information for Bframe 302 is predicted (here, e.g., interpolated) from the motion vectorinformation encoded for P frames 300 and 304. The exemplary technique isderived from the assumption that an object is moving with constantspeed, and thus making it possible to predict its current positioninside B frame 302 without having to transmit any motion vectors. Whilethis technique may reduce the bitrate significantly for a given quality,it may not always be applied.

Introduced herein, in accordance with certain implementations of thepresent invention, is a new Inter Macroblock type is provided that caneffectively exploit spatial and temporal correlations that may exist atthe macroblock level and in particular with regard to the motion vectorinformation of the macroblock. According to this new mode it is possiblethat a current macroblock may have motion that can be directly derivedfrom previously decoded information (e.g., Motion Projection). Thus, asillustratively shown in FIG. 4, there may not be a need to transmit anymotion vectors for a macroblock, but even for an entire frame. Here, asequence 400 of video frames is depicted with solid arrows indicatingcoded relationships between frames and dashed lines indicatingpredictable macroblock relationships. Video frame 402 is an I frame,video frames 404, 406, 410, and 412 are B frames, and video frames 408and 414 are P frames. In this example, if P frame 408 has a motion fielddescribed by {right arrow over (MF)}₄₀₆ the motion of the collocatedmacroblocks in pictures 404, 406, and 414 is also highly correlated. Inparticular, assuming that speed is in general constant on the entireframe and that frames 404 and 406 are equally spaced in time betweenframes 402 and 408, and also considering that for B frames both forwardand backward motion vectors could be used, the motion fields in frame404 could be equal to {right arrow over (MF)}₄₀₄ ^(fw)=⅓×{right arrowover (MF)}₄₀₆ and {right arrow over (MF)}₄₀₄ ^(bw)=−⅔×{right arrow over(MF)}₄₀₆ for forward and backward motion fields respectively. Similarly,for frame 408 the motion fields could be {right arrow over (MF)}₄₀₈^(fw)=⅔×{right arrow over (MF)}₄₀₆ and {right arrow over (MF)}₄₀₈^(bw)=−⅓×{right arrow over (MF)}₄₀₆ for forward and backward motionvectors respectively. Since 414 and 406 are equally spaced, then, usingthe same assumption, the collocated macroblock could have motion vectors{right arrow over (MF)}₄₁₆ ={right arrow over (MF)}₄₀₆.

Similar to the Direct Mode in B frames, by again assuming that speed isconstant, motion for a macroblock can be directly derived from thecorrelated macroblock of the reference frame. This is furtherillustrated in FIG. 6, for example, which shows three video frames,namely a P frame 600, a B frame 602 and P frame 604, corresponding totimes t, t+1, and t+2, respectively. Here, the illustrated collocatedmacroblocks have similar if not identical motion information.

It is even possible to consider acceleration for refining such motionparameters, for example, see FIG. 7. Here, for example, three frames areshown, namely a current frame 704 at time t, and previous frames 702(time t−1) and 700 (time t−2), with different acceleration informationillustrated by different length motion vectors.

The process may also be significantly improved by, instead ofconsidering motion projection at the macroblock level, taking intoaccount that the pixels inside the previous image are possibly movingwith a constant speed or a constant acceleration (e.g., PixelProjection). As such, one may generate a significantly more accurateprediction of the current frame for B frame coding as illustrated, forexample, in FIG. 8, and for P frame coding as illustrated, for example,in FIG. 9. FIG. 8, for example, shows three video frames, namely a Pframe 800, a B frame 802 and P frame 804, corresponding to times t, t+1,and t+2, respectively. FIG. 9, for example, shows three video frames,namely a P frame 900, a B frame 902 and P frame 904, corresponding totimes t, t+1, and t+2, respectively.

In certain implementations it is also possible to combine both methodstogether for even better performance.

In accordance with certain further implementations, motion can also bederived from spatial information, for example, using predictiontechniques employed for the coding of motion vectors from the motioninformation of the surrounding macroblocks. Additionally, performancecan also be further enhanced by combining these two different methods ina multi-hypothesis prediction architecture that does not require motioninformation to be transmitted. Consequently, such new macroblock typescan achieve significant bitrate reductions while achieving similar orimproved quality.

Exemplary Encoding Processes:

FIG. 10 illustrates an exemplary encoding environment 1000, having aconventional block based video encoder 1002, wherein a video data 1004is provided to encoder 1002 and a corresponding encoded video databitstream is output.

Video data 1004 is provided to a summation module 1006, which alsoreceives as an input, the output from a motion compensation (MC) module1022. The output from summation module 1006 is provided to a discretecosine transform (DCT) module 1010. The output of DCT module 1010 isprovided as an input to a quantization module (QP) 1012. The output ofQP module 1012 is provided as an input to an inverse quantization module(QP⁻¹) 1014 and as an input to a variable length coding (VLC) module1016. VLC module 1016 also receives as in input, an output from a motionestimation (ME) module 1008. The output of VLC module 1016 is an encodedvideo bitstream 1210.

output of QP⁻¹ module 1014 is provided as in input to in inversediscrete cosine transform (DCT) module 1018. The output of 1018 isprovided as in input to a summation module 1020, which has as anotherinput, the output from MC module 1022. The output from summation module1020 is provided as an input to a loop filter module 1024. The outputfrom loop filter module 1024 is provided as an input to a frame buffermodule 1026. One output from frame buffer module 1026 is provided as aninput to ME module 1008, and another output is provided as an input toMC module 1022. Me module 1008 also receives as an input video data1004. An output from ME 1008 is proved as an input to MC module 1022.

In this example, MC module 1022 receives inputs from ME module 1008.Here, ME is performed on a current frame against a reference frame. MEcan be performed using various block sizes and search ranges, afterwhich a “best” parameter, using some predefined criterion for example,is encoded and transmitted (INTER coding). The residue information isalso coded after performing DCT and QP. It is also possible that in somecases that the performance of ME does not produce a satisfactory result,and thus a macroblock, or even a subblock, could be INTRA encoded.

Considering that motion information could be quite costly, the encodingprocess can be modified as in FIG. 12, in accordance with certainexemplary implementations of the present invention, to also consider ina further process the possibility that the motion vectors for amacroblock could be temporally and/or spatially predicted frompreviously encoded motion information. Such decisions, for example, canbe performed using Rate Distortion Optimization techniques or other costmeasures. Using such techniques/modes it may not be necessary totransmit detailed motion information, because such may be replaced witha Direct Prediction (Direct P) Mode, e.g., as illustrated in FIG. 5.

Motion can be modeled, for example, in any of the following models ortheir combinations: (1) Motion Projection (e.g., as illustrated in FIG.3 for B frames and FIG. 6 for P frames); (2) Pixel Projection (e.g., asillustrated in FIG. 8 for B frames and FIG. 9 for P frames); (3) SpatialMV Prediction (e.g., median value of motion vectors of collocatedmacroblocks); (4) Weighted average of Motion Projection and SpatialPrediction; (5) or other like techniques.

Other prediction models (e.g. acceleration, filtering, etc.) may also beused. If only one of these models is to be used, then this should becommon in both the encoder and the decoder. Otherwise, one may usesubmodes which will immediately guide the decoder as to which model itshould use. Those skilled in the art will also recognize thatmulti-referencing a block or macroblock is also possible using anycombination of the above models.

In FIG. 12, an improved video encoding environment 1200 includes a videoencoder 1202 that receives video data 1004 and outputs a correspondingencoded video data bitstream.

Here, video encoder 1202 has been modified to include improvement 1204.Improvement 1204 includes an additional motion vector (MV) buffer module1206 and a DIRECT decision module 1208. More specifically, as shown, MVbuffer module 1206 is configured to receive as inputs, the output fromframe buffer module 1026 and the output from ME module 1008. The outputfrom MV buffer module 1206 is provided, along with the output from MEmodule 1008, as an input to DIRECT decision module 1208. The output fromDIRECT decision module 1208 is then provided as an input to MC module1022 along with the output from frame buffer module 1026.

For the exemplary architecture to work successfully, the MotionInformation from the previously coded frame is stored intact, which isthe purpose for adding MV buffer module 1206. MV buffer module 1206 canbe used to store motion vectors. In certain implementations. MV buffermodule 1206 may also store information about the reference frame usedand of the Motion Mode used. In the case of acceleration, for example,additional buffering may be useful for storing motion information of the2^(nd) or even N previous frames when, for example, a more complicatedmodel for acceleration is employed.

If a macroblock, subblock, or pixel is not associated with a MotionVector (i.e., a macroblock is intra coded), then for such block it isassumed that the Motion Vector used is (0, 0) and that only the previousframe was used as reference.

If multi-frame referencing is used, one may select to use the motioninformation as is, and/or to interpolate the motion information withreference to the previous coded frame. This is essentially up to thedesign, but also in practice it appears that, especially for the case of(0, 0) motion vectors, it is less likely that the current block is stillbeing referenced from a much older frame.

One may combine Direct Prediction with an additional set of MotionInformation which is, unlike before, encoded as part of the DirectPrediction. In such a case the prediction can, for example, be amulti-hypothesis prediction of both the Direct Prediction and the MotionInformation.

Since there are several possible Direct Prediction submodes that one maycombine, such could also be combined within a multi-hypothesisframework. For example, the prediction from motion projection could becombined with that of pixel projection and/or spatial MV prediction.

Direct Prediction can also be used at the subblock level within amacroblock. This is already done for B frames inside the current H.26Lcodec, but is currently only using Motion Projection and not PixelProjection or their combinations.

For B frame coding, one may perform Direct Prediction from only onedirection (forward or backward) and not always necessarily from bothsides. One may also use Direct Prediction inside the Bidirectional modeof B frames, where one of the predictions is using Direct Prediction.

In the case of Multi-hypothesis images, for example, it is possible thata P frame is referencing to a future frame. Here, proper scaling, and/orinversion of the motion information can be performed similar to B framemotion interpolation.

Run-length coding, for example, can also be used according to which, ifsubsequent “equivalent” Direct P modes are used in coding a frame orslice, then these can be encoded using a run-length representation.

DIRECT decision module 1208 essentially performs the decision whetherthe Direct Prediction mode should be used instead of the pre-existingInter or Intra modes. By way of example, the decision may be based onjoint Rate/Distortion Optimization criteria, and/or also separatebitrate or distortion requirements or restrictions.

It is also possible, in alternate implementations, that module DirectPrediction module 1208 precedes the ME module 1008. In such case, ifDirect Prediction can provide immediately with a good enough estimate,based on some predefined conditions, for the motion parameters, MEmodule 1008 could be completely by-passed, thus also considerablyreducing the computation of the encoding.

Exemplary Decoding Processes:

Reference is now made to FIG. 11, which depicts an exemplaryconventional decoding environment 1100 having a video decoder 1102 thatreceives an encoded video data bitstream 1104 and outputs corresponding(decoded) video data 1120.

Encoded video data bitstream 1104 is provided as an input to a variablelength decoding (VLD) module 1106. The output of VLD module 1106 isprovided as an input to a QP⁻¹ module 1108, and as an input to an MCmodule 1110. The output from QP⁻¹ module 1108 is provided as an input toan IDCT module 1112. The output of IDCT module 1112 is provided as aninput to a summation module 1114, which also receives as an input anoutput from MC module 1110. The output from summation module 1114 isprovided as an input to a loop filter module 1116. The output of loopfilter module 1116 is provided to a frame buffer module 1118. An outputfrom frame buffer module 1118 is provided as an input to MC module 1110.Frame buffer module 1118 also outputs (decoded) video data 1120.

An exemplary improved decoder 1302 for use in a Direct Predictionenvironment 1300 further includes an improvement 1306. Here, as shown inFIG. 13, improved decoder 1302 receives encoded video data bitstream1210, for example, as output by improved video encoder 1202 of FIG. 12,and outputs corresponding video (decoded) video data 1304.

Improvement 1306, in this example, is operatively inserted between MCmodule 1110 and a VLD module 1106′. Improvement 1306 includes an MVbuffer module 1308 that receives as an input, an output from VLD module1106′. The output of MV buffer module 1308 is provided as a selectableinput to a selection module 1312 of improvement 1306. A block modemodule 1318 is also provided in improvement 1306. Block mode module 1318receives as an input, an output from VLD module 1106′. An output ofblock mode module 1318 is provided as an input to VLD module 1106′, andalso as a controlling input to selection module 1312. An output from VLDmodule 1106′ is provided as a selectable input to selection module 1312.Selection module 1312 is configured to selectably provide either anoutput from MV buffer module 1308 or VLD module 1106′ as an input to MCmodule 1110.

With improvement 1306, for example, motion information for each pixelcan be stored, and if the mode of a macroblock is identified as theDirect Prediction mode, then the stored motion information, and theproper Projection or prediction method is selected and used. It shouldbe noted that if Motion Projection is used only, then the changes in anexisting decoder are very minor, and the additional complexity that isadded on the decoder could be considered negligible.

If submodes are used, then improved decoder 1302 can, for example, beconfigured to perform steps opposite to the prediction steps thatimproved encoder 1202 performs, in order to properly decode the currentmacroblock.

Again non referenced pixels (such as intra blocks) may be considered ashaving zero motion for the motion storage.

Some Exemplary Schemes

Considering that there are several possible predictors that may beimmediately used with Direct Prediction, for brevity purposes in thisdescription a smaller subset of cases, which are not only ratherefficient but also simple to implement, are described in greater detail.In particular, the following models are examined in greaterdemonstrative detail:

(A) In this example, Motion Projection is the only mode used. Norun-length coding of Direct Modes is used, where as residue informationis also transmitted. A special modification of the motion parameters isperformed in the case that a zero motion vector is used. In such asituation, the reference frame for the Direct Prediction is always setto zero (e.g., previous encoded frame). Furthermore, intra coded blocksare considered as having zero motion and reference frame parameters.

-   -   (B) This example is like example (A) except that no residue is        transmitted.    -   (C) This example is basically a combination of examples (A) and        (B), in that if QP<n (e.g., n=24) then the residue is also        encoded, otherwise no residue is transmitted.    -   (D) This example is an enhanced Direct Prediction scheme that        combines three submodes, namely:        -   (1)Motion Projection ({right arrow over (MV)}_(MP));        -   (2) Spatial MV Prediction ({right arrow over (MV)}_(SP));            and        -   (3) A weighted average of these two cases

$( \frac{\lbrack {{\overset{arrow}{MV}}_{MP} + {2*{\overset{arrow}{MV}}_{SP}}} \rbrack}{3} ).$Wherein, residue is not transmitted for QP<n (e.g., n=24). Here,run-length coding is not used. The partitioning of the submodes can beset as follows:

Submodes Code Spatial Predictor 0 Motion Projection 1 Weighted Average 2

The best submode could be selected using a Rate Distortion Optimizationprocess (best compromise between bitrate and quality).

-   -   (E) A combination of example (C) with Pixel Projection. Here,        for example, an average of two predictions for the Direct        Prediction Mode.    -   (F) This is a combination of example (C) with Motion_Copy R2        (see. e.g., Jani Lainema and Marta Karczewicz, “Skip mode motion        compensation”, Doc. JVT-C027, May 2002, which is incorporated        herein by reference) or the like. This case can be seen as an        alternative of the usage of the Spatial MV Predictor used in        example (D), with one difference being that the spatial        predictor, under certain conditions, completely replaces the        zero skip mode, and that this example (F) can be run-length        encoded thus being able to achieve more efficient performance.

Motion Vector Prediction in Bidirectionally Predictive (B) Frames withRegards to Direct Mode:

The current JVT standard appears to be quite unclear on how a DirectMode coded macroblock or block should be considered in the motion vectorprediction within Bidirectionally Predicted (B) frames. Instead, itappears that the current software considers a Direct Mode Macroblock orsubblock as having a “different reference frame” and thus not used inthe prediction. Unfortunately, considering that there might still behigh correlation between the motion vectors of a Direct predicted blockwith its neighbors such a condition could considerably hinder theperformance of B frames and reduce their efficiency. This could alsoreduce the efficiency of error concealment algorithms when applied to Bframes.

In this section, exemplary alternative approaches are presented, whichcan improve the coding efficiency increase the correlation of motionvectors within B frames, for example. This is done by considering aDirect Mode coded block essentially equivalent to a Bidirectionallypredicted block within the motion prediction phase.

Direct Mode Macroblocks or blocks (for example, in the case of 8×8sub-partitions) could considerably improve the efficacy ofBidirectionally Predicted (B) frames since they can effectively exploittemporal correlations of motion vector information of adjacent frames.The idea is essentially derived from temporal interpolation techniqueswhere the assumption is made that if a block has moved from a position(x+dx, y+dy) at time t to a position (x, y) at time t+2, then, by usingtemporal interpolation, at time t+1 the same block must have essentiallybeen at position:

$( {{x + \frac{dx}{2}},{y + \frac{dy}{2}}} )$

This is illustrated, for example, in FIG. 14, which shows three frames,namely, a P frame 1400, a B frame 1402 and P frame 1404, correspondingto times t, t+1, and t+2, respectively. The approach though most oftenused in current encoding standards instead assumes that the block atposition (x, y) of frame at time t+1 most likely can be found atpositions:

${( {{x + \frac{dx}{2}},{y + \frac{dy}{2}}} )\mspace{11mu}{at}\mspace{14mu}{time}\mspace{14mu} t\mspace{14mu}{and}\mspace{11mu}( {{x - \frac{dx}{2}},{y - \frac{dy}{2}}} )\mspace{11mu}{at}\mspace{14mu}{time}\mspace{14mu} t} + 2.$

The later is illustrated in FIG. 15, which shows three frames, namely, aP frame 1500, a B frame 1502 and P frame 1504, corresponding to times t,t+1, and t+2, respectively. Since the number of Direct Mode coded blockswithin a sequence can be significant, whereas no residue and motioninformation are transmitted for such a case, efficiency of B frames canbe considerably increased. Run-length coding (for example, if theUniversal Variable Length Code (UVLC) entropy coding is used) may alsobe used to improve performance even further.

Unfortunately, the current JVT standard does not clarify how the motionvector prediction of blocks adjacent to Direct Mode blocks should beperformed. As it appears from the current software, Direct Mode blocksare currently considered as having a “different reference frame” thus nospatial correlation is exploited in such a case. This could considerablyreduce the efficiency of the prediction, but could also potentiallyaffect the performance of error concealment algorithms applied on Bframes in case such is needed.

By way of example, if one would like to predict the motion vector of Ein the current codec, if A, B, C, and D were all Direct Mode coded, thenthe predictor will be set as (0,0) which would not be a good decision.

In FIG. 16, for example, E is predicted from A, B, C, and D. Thus, if A,B, C, or D are Direct Mode coded then their actual values are notcurrently used in the prediction. This can be modified, however. Thus,for example, if A, B, C, or D are Direct Mode coded, then actual valuesof Motion Vectors and reference frames can be used in the prediction.This provides two selectable options: (1) if collocated macroblock/blockin subsequent P frame is intra coded then a reference frame is set to−1; (2) if collocated macroblock/block in subsequent P frame is intracoded then assume reference frame is 0.

In accordance with certain aspects of the present invention, instead onemay use the actual Motion information available from the Direct Modecoded blocks, for performing the motion vector prediction. This willenable a higher correlation of the motion vectors within a B framesequence, and thus can lead to improved efficiency.

One possible issue is how to appropriately handle Direct ModeMacroblocks for which, the collocated block/macroblock in the subsequentframe was intra coded. Here, for example, two possible options include:

-   -   (1) Consider this macroblock/block as having a different        reference frame, thus do not use it in the motion vector        prediction; and    -   (2) Consider this macroblock as having (0, 0) motion vector and        reference frame 0.

In accordance with certain other exemplary implementations of thepresent invention, a further modification can be made in the de-blockingfilter process. For the Direct Mode case, a de-blocking filter processcan be configured to compare stored motion vector information that istaken from Direct Mode coded blocks—otherwise these would usually beconsidered as zero. In another modification, however, instead one mayconfigure the de-blocking filter process to compare the (exact) motionvectors regardless of the block type that is used. Thus, in certainimplementations, if for Direct Coded blocks no residue is transmitted, a“stronger” de-blocking filter can provide further improved performance.

Furthermore, in certain other implementations, the Rate DistortionDecision for B frames can be redesigned since it is quite likely thatfor certain implementations of the motion vector prediction scheme, adifferent langrangian parameter λ used in Rate Distortion Optimizationdecisions, may lead to farther coding efficiency. Such λ can be taken,for example, as:

$\lambda = {0.85 \times 2^{\frac{Qp}{3}}}$Inter Mode Decision Refinement:

The JVT standard currently has an overwhelming performance advantageversus most other current Block Based coding standards. Part of thisperformance can be attributed in the possibility of using variable blocksizes raging from 16×16 down to 4×4 (pixels), instead of having fixedblock sizes. Doing so, for example, allows for a more effectiveexploitation of temporal correlation. Unfortunately, it has been foundthat, due to the Mode Decision techniques currently existing inconventional coding logic (e.g., hardware, firmware, and/or software),mode decisions might not be optimally performed, thus wasting bits thatcould be better allocated.

In this section, further methods and apparatuses are provided that atleast partly solve this problem and/or others. Here, the exemplarymethods and apparatuses have been configured for use with at least 16×8and 8×16 (pixel) block modes. Furthermore, using a relatively simplesolution where at least one additional criterion is introduced, a savingof between approximately 5% and 10% is provided in the complexity of theencoder.

Two key features of the JVT standard are variable macroblock modeselection and Rate Distortion Optimization. A 16×16 (pixel) macroblockcan be coded using different partitioning modes for which motioninformation is also transmitted. The selection of the mode to be usedcan be performed in the Rate Distortion Optimization phase of theencoding where a joint decision of best possible quality at bestpossible bitrate is attempted. Unfortunately, since the assignments ofthe best possible motion information for each subpartition is done in anentirely different process of the encoding, it is possible in somecases, that a non 16×16 mode (e.g. 16×8 or 8×16 (pixel)) carries motioninformation that is equivalent to a 16×16 macroblock. Since the motionpredictors used for each mode could also be different, it is quitepossible in many cases that such 16×16 type motion information could bedifferent from the one assigned to the 16×16 mode. Furthermore, undercertain conditions, the Rate Distortion Optimization may in the enddecide to use the non 16×16 macroblock type, even though it continues16×16 motion information, without examining whether such could have beenbetter if coded using a 16×16 mode.

Recognizing this, an exemplary system can be configured to determinewhen such a case occurs, such that improved performance may be achieved.In accordance with certain exemplary implementations of the presentinvention, two additional modes, e.g., referred to as P2to1 and P3to1,are made available within the Mode decision process/phase. The P2to1 andP3to1 modes are enabled when the motion information of a 16×8 and 8×16subpartitioning, respectively, is equivalent to that of a 16×16 mode.

In certain implementations all motion vectors and reference frameassigned to each partition may be equal. As such, the equivalent modecan be enabled and examined during a rate distortion process/phase.Since the residue and distortion information will not likely changecompared to the subpartition case, they can be reused withoutsignificantly increasing computation.

Considering though that the Rate Distortion Mode Decision is notperfect, it is possible that the addition and consideration of these twoadditional modes regardless of the current best mode may, in somelimited cases, reduce the efficiency instead of improving it. As analternative, one may enable these modes only when the correspondingsubpartitioning mode was also the best possible one according to theMode decision employed. Doing so may yield improvements (e.g., bitratereduction) versus the other logic (e.g., codecs, etc.), while notaffecting the PSNR.

If the motion information of the 16×8 or 8×16 subpartitioning isequivalent to that of the 16×16 mode, then performing mode decision forsuch a mode may be unnecessary. For example, if the motion vectorpredictor of the first subpartition is exactly the same as the motionvector predictor of the 16×16 mode performing mode decision isunnecessary. If such condition is satisfied, one may completely skipthis mode during the Mode Decision process. Doing so can significantlyreduce complexity since it would not be necessary, for this mode, toperform DCT, Quantization, and/or other like Rate Distortionprocesses/measurements, which tend to be rather costly during theencoding process.

In certain other exemplary implementations, the entire process can befurther extended to a Tree-structured macroblock partition as well. See,e.g., Heiko Schwarz and Thomas Wiegand, “Tree-structured macroblockpartition”, Doc. VCEG-N17, December 2001.

An Exemplary Algorithm

Below are certain acts that can be performed to provide a moderefinement in an exemplary codec or other like logic (note that incertain other implementations, the order of the act may be changedand/or that certain acts may be performed together):

-   -   Act 1: Set Valid[P2to1]=Valid [P3to1]=0.    -   Act 2: Perform Motion Vector and Reference frame decision for        each possible Inter Mode. Let {right arrow over (MV)}_(16×16),        {right arrow over (MVP)}_(16×16), and refframe_(16×16) be the        motion vector, motion vector predictor, and reference frame of        the 16×16 mode, {{right arrow over (MV^(a))}_(16×8), {right        arrow over (MV^(b))}_(16×8)}, {{right arrow over        (MVP^(a))}_(16×8),{right arrow over (MVP^(b))}_(16×8)}, and        {refframe_(16×8) ^(a), refframe_(16×8) ^(b)} the corresponding        information for the 16×8 mode, and {{right arrow over        (MV^(a))}_(8×16), {right arrow over (MV^(b))}_(8×16)}, {{right        arrow over (MVP^(a))}_(8×16), {right arrow over        (MVP^(b))}_(8×16)}, and {refframe_(8×16) ^(a), refframe_(8×16)        ^(b) } for the 8×16 mode.    -   Act 3: If ({right arrow over (MV^(a))}_(16×8)!={right arrow over        (MV^(b))}_(16×8))OR(refframe_(16×8) ^(a)!=refframe_(16×8) ^(b))        and goto Act 7.    -   Act 4: If ({right arrow over (MV^(a))}_(16×8)!={right arrow over        (MV)}_(16×16))OR({right arrow over (MVP^(a))}_(16×8)!={right        arrow over (MVP)}_(16×16) )OR(refframe_(16×8)        ^(a)!=refframe_(16×16)), then goto Act 6.    -   Act 5: Valid[16×8 ]=0; goto Act 7 (e.g.., Disable 16×8 mode if        identical to 16×16. Complexity reduction).    -   Act 6: Valid[P2to1]=1; (e.g., Enable refinement mode for 16×8)        {right arrow over (MV)}_(P2to1)={right arrow over        (MV^(a))}_(16×8); refframe_(P2to1)=refframe_(16×8) ^(a);    -   Act 7: If ({right arrow over (MV^(a))}_(8×16)!={right arrow over        (MV^(b))}_(8×16))OR(refframe_(8×16) ^(a)!=refframe_(8×16) ^(b)),        then goto Act 11.    -   Act 8: If ({right arrow over (MV^(a))}_(8×16) !={right arrow        over (MV)}_(16×16))OR({right arrow over        (MVP^(a))}_(8×16)!={right arrow over        (MVP)}_(16×16))OR(refframe_(8×16) ^(a)!=refframe_(16×16)) then        goto Act 10.    -   Act 9: Valid[8×16]=0; goto Act 11 (e.g., Disable 8×16 mode if        identical to 16×16 to reduce complexity)    -   Act 10: Valid[P3to1]=1 (e.g., enable refinement mode for 8×16)        {right arrow over (MV)}_(P3to1)={right arrow over        (MV^(a))}_(8×16); refframe_(P3to1)=refframe_(8×16) ^(a);    -   Act 11: Perform Rate Distortion Optimization for all Inter &        Intra modes if (Valid[MODE]=1) where MODE ε{INTRA 4×4, INTRA        16×16, SKIP, 16×16, 16×8, 8×16, P8×8}, using the langrangian        functional: J(s, c, MODE|QP, λ_(MODE))=SSD(s, c,        MODE|QP)+λ_(MODE)·R (s,c,MODE |Qp) ActSet best mode to BestMode    -   Act 12: If (BestMode!=16×8) then Valid[P3to1]=0 (note that this        act is optional).    -   Act 13 If (BestMode!=8×16) then Valid [P2to1]=0 (note that this        act is optional).    -   Act 14: Perform Rate Distortion Optimization for the two        additional modes if (Valid[MODE]=1) where MODE ε{P2to1, P3to1}        (e.g., modes are considered equivalent to 16×16 modes).    -   Act 15: Set BestMode to the overall best mode found.        Applying Exempla Direct Prediction Techniques For Interlace        Coding:

Due to the increased interest of interlaced video coding inside theH.26L standard, several proposals have been presented on enhancing theencoding performance of interlaced sequences, In this section techniquesare presented that can be implemented in the current syntax of H.26L,and/or other like systems. These exemplary techniques can provideperformance enhancement. Furthermore, Direct P Prediction technology isintroduced, similar to Direct B Prediction, which can be applied in bothinterlaced and progressive video coding.

Further Information On Exemplary Direct P Prediction Techniques:

Direct Mode of motion vectors inside B-frames can significantly benefitencoding performance since it can considerably reduce the bits requiredfor motion vector encoding, especially considering that up to two motionvectors have to be transmitted. If, though, a block is coded usingDirect Mode, no motion vectors are necessary where as instead these arecalculated as temporal interpolations of the motion vectors of thecollocated blocks in the first subsequent reference image. A similarapproach for P frames appears to have never been considered since thestructure of P frames and of their corresponding macroblock was muchsimpler, while each macroblock required only one motion vector. Addingsuch a mode would have instead, most likely, incurred a significantoverhead, thus possibly negating any possible gain.

In H.26L on the other hand, P frames were significantly enhanced byintroducing several additional macroblock Modes. As describedpreviously, in many cases it might even be necessary to transmit up to16 motion vectors per macroblock. Considering this additional ModeOverhead that P frames in H.26L may contain, an implementation of DirectPrediction of the motion vectors could be viable. In such a way, allbits for the motion vectors and for the reference frame used can besaved at only the cost of the additional mode, for example, see FIG. 4.

Even though a more straightforward method of Direct P prediction is toselect the Motion vectors of the collocated pixels in the previousframe, in other implementations one may also consider MotionAcceleration as an alternative solution. This comes from the fact thatmaybe motion is changing frame by frame, it is not constant, and byusing acceleration better results could be obtained, for example, seeFIG. 7.

Such techniques can be further applied to progressive video coding.Still, considering the correlation that fields may have in some casesinside interlace sequences, such as for example regions with constanthorizontal only movement, this approach can also help improve codingefficiency for interlace sequence coding. This is in particularbeneficial for known field type frames, for example, if it is assumedthat the motion of adjacent fields is the same. In this type ofarrangement, same parity fields can be considered as new frames and aresequentially coded without taking consideration of the interlacefeature. Such is entirely left on the decoder. By using this exemplaryDirect P mode though, one can use one set of motion vectors for thefirst to be coded field macroblock (e.g., of size 16×16 pixels) where asthe second field at the same location is reusing the same motioninformation. The only other information necessary to be sent is thecoded residue image. In other implementations, it is possible to furtherimprove upon these techniques by considering correlations between theresidue images of the two collocated field Blocks.

In order to allow Direct Mode in P frames, it is basically necessary toadd one additional Inter Mode into the system Thus, instead of havingonly 8 Inter Modes, in one example, one can now use 9 which are shownbelow:

INTER MODES Description COPY_MB 0 Skip macroblock Mode M16×16_MB 1 One16 × 16 block M16×8_MB 2 Two 16 × 8 blocks M8×16_MB 3 Two 8 × 16 blocksM8×8_MB 4 Four 8 × 8 blocks M8×4_MB 5 Eight 8 × 4 blocks M4×8_MB 6 Eight4 × 8 blocks M4×4_MB 7 Sixteen 16 × 8 blocks PDIRECT_MB 8 Copy Mode andmotion vectors of collocated macroblock in previous frame

In general, such exemplary Direct Modes for P frames can appear if thecollocated macroblock was also of INTER type, except Skip macroblock,but including Direct Mode, since in other cases there is no motioninformation that could be used. In the case of the previous macroblockalso being coded in Direct P Mode, the most recent Motion Vectors andMode for this macroblock are considered instead. To more efficientlythough handle the cases that this Mode will not logically appear, and inparticular if INTRA mode was used, one may select of allowing this Modeto also appear in such cases with the Mode now signifying a second SkipMacroblock Mode where a copy the information is not from the previousframe, but from the one before it. In this case, no residue informationis encoded, This is particularly useful for Interlace sequences, sinceit is more likely that a macroblock can be found with higher accuracyfrom the same parity field frame, and not from the previously codedfield frame as was presented in previous techniques.

For further improved efficiency, if a set of two Field type frames isused when coding interlace images, the Skip Macroblock Mode can beconfigured to use the same parity field images. If Direct P mode is usedas a skipping flag, for example, then the different parity is usedinstead. An additional benefit of Direct P mode, is that it may allowfor a significant complexity reduction in the encoder since it ispossible to allow the system to perform a pre-check to whether theDirect P mode gives a satisfactory enough solution, and if so, noadditional computation may be necessary for the mode decision and motionestimation of that particular block. To also address the issue of motionvector coding, the motion vectors used for Direct P coding can be used“as is” for the calculation of a MEDIAN predictor.

Best Field First technique & Field Reshuffling:

Coding of interlaced sequence allowing support of both interlace framematerial, and separate interlace field images inside the same streamwould likely provide a much better solution than coding using only oneof the two methods. The separate interlace field technique has someadditional benefits, such as, for example, de-blocking, and inparticular can provide enhanced error resilience. If an error happensinside one field image, for example, the error can be easily consumedusing the information from the second image.

This is not the case for the frame based technique, where especiallywhen considering the often large size of and bits used by such frames,errors inside such a frame can happen with much higher probability.Reduced correlation between pixels/blocks may not promote errorrecovery.

Here, one can further improve on the field/frame coding concept byallowing the encoder to select which field should be encoded first,while disregarding which field is to be displayed first. This could behandled automatically on a decoder where a larger buffer will be neededfor storing a future field frame before displaying it. For example, eventhough the top field precedes the bottom field in terms of time, thecoding efficiency might be higher if the bottom field is coded andtransmitted first, followed by the top field frame. The decision may bemade, for example, in the Rate Distortion Optimization process/phase,where one first examines what will the performance be if the Odd fieldis coded first followed by the Even field, and of the performance if theEven field is instead coded and is used as a reference for the Oddfield. Such a method implies that both the encoder and the decodershould know which field should be displayed first, and any reshufflingdone seamlessly. It is also important that even though the Odd field wascoded first, both encoder and decoder are aware of this change whenindexing the frame for the purpose of INTER/INTRA prediction.Illustrative examples of such a prediction scheme, using 4 referenceframes, are depicted in FIG. 17 and FIG. 18. In FIG. 17, interlacecoding is shown using an exemplary Best Field First scheme in P frames.In FIG. 18, interlace coding is shown using a Best Field First scheme inB frames.

In the case of coding joint field/frame images, the schemeillustratively depicted in FIG. 19 may be employed. Here, an exemplaryimplementation of a Best Field First scheme with frame and field basedcoding is shown. If two frames are used for the frame based motionestimation, then at least five field frames can be used for motionestimation of the fields, especially if field swapping occurs. Thisallows referencing of at least two field frames of the same parity. Ingeneral 2×N+1 field frames should be stored if N full frames are to beused. Frames also could easily be interleaved and deinterleaved on theencoder and decoder for such processes.

CONCLUSION

Although the description above uses language that is specific tostructural features and/or methodological acts, it is to be understoodthat the invention defined in the appended claims is not limited to thespecific features or acts described. Rather, the specific features andacts are disclosed as exemplary forms of implementing the invention.

1. In a device that implements a video encoder, the device including aprocessor and memory, a method of encoding video data within a sequenceof video frames, the method comprising: with the device that implementsthe video encoder, encoding at least a portion of a predictive (P) framerelative to at least one reference frame to include motion informationassociated with the portion of the P frame relative to the at least onereference frame; with the device that implements the video encoder,defining a first direct mode portion of a bidirectionally predictive (B)frame that includes video data predictively correlated to the at leastone reference frame based on the motion information associated with theportion of the P frame; with the device that implements the videoencoder, defining a second direct mode portion of the B frame thatincludes video data predictively correlated to the at least onereference frame based on motion information determined using spatialmotion vector prediction with motion information for other portions inthe B frame; and with the device that implements the video encoder,encoding the first and second direct mode portions of the B framewithout including corresponding motion information for the first andsecond direct mode portions and including mode identifying data thatidentifies that (a) the first direct mode portion of the B frame can bedirectly derived using at least the motion information associated withthe portion of the P frame relative to the at least one reference frameand (b) the second direct mode portion of the B frame can be directlyderived using the motion information determined using spatial motionvector prediction with motion information for other portions in the Bframe.
 2. The method of claim 1, wherein the mode identifying datadefines a type of prediction model required to decode the encoded firstand second direct mode portions of the B frame.
 3. The method of claim1, wherein the portion of the P frame is a macroblock, and wherein thefirst and second direct mode portions of the B frame are macroblocks. 4.The method of claim 1, wherein the motion information associated withthe portion of the P frame relative to the at least one reference frameincludes Pixel Projection information required to decode the encodedfirst portion of the B frame.
 5. The method of claim 1, furthercomprising: with the device that implements the video encoder,selectively determining if a direct prediction mode is used instead of apre-existing mode for the other portions of the B frame.
 6. In a devicethat implements a video encoder, the device including a processor andmemory, a method comprising: with the device that implements the videoencoder, encoding video for a sequence, including: encoding a firstvideo frame in the sequence, wherein at least some of the first videoframe uses motion compensation relative to a reference frame; encoding asecond video frame in the sequence, wherein at least some of the secondvideo frame uses bi-directionally predictive motion compensation,including: determining whether one or more direct mode macroblocks ofthe second video frame use (a) motion vectors based upon motioninformation associated with macroblocks of the first video frame or (b)motion vectors determined according to spatial motion vector predictionwith motion information of other macroblocks in the second video frame;for the one or more direct mode macroblocks of the second video frame,using spatial motion vector prediction to determine the motion vectorsfrom the motion information of other macroblocks in the second videoframe; and encoding the one or more direct mode macroblocks of thesecond video frame using the motion vectors determined using spatialmotion vector prediction; and with the device that implements the videoencoder, outputting the encoded video in a bit stream.
 7. The method ofclaim 6 wherein the first video frame is a P frame and the second videoframe is a B frame.
 8. The method of claim 6 wherein the outputtingincludes outputting mode identifying data in the bit stream to indicateto a decoder whether to use motion projection or spatial motion vectorprediction for the one or more direct mode macroblocks, and wherein themotion projection uses the motion information associated withmacroblocks of the first video frame.
 9. A video encoder comprising: aprocessor; memory; and storage storing computer-executable instructionsfor causing the video encoder to perform a method comprising: encodingvideo for a sequence, including: encoding a first video frame in thesequence, wherein at least some of the first video frame uses motioncompensation relative to a reference frame; encoding a second videoframe in the sequence, wherein at least some of the second video frameuses bi-directionally predictive motion compensation, including:determining whether one or more direct mode macroblocks of the secondvideo frame use (a) motion vectors based upon motion informationassociated with macroblocks of the first video frame or (b) motionvectors determined according to spatial motion vector prediction withmotion information of other macroblocks in the second video frame; forthe one or more direct mode macroblocks of the second video frame, usingspatial motion vector prediction to determine the motion vectors fromthe motion information of other macroblocks in the second video frame;and encoding the one or more direct mode macroblocks of the second videoframe using the motion vectors determined using spatial motion vectorprediction; and outputting the encoded video in a bit stream.
 10. Thevideo encoder of claim 9 wherein the first video frame is a P frame andthe second video frame is a B frame.
 11. The video encoder of claim 9wherein the outputting includes outputting mode identifying data in thebit stream to indicate to a decoder whether to use motion projection orspatial motion vector prediction for the one or more direct modemacroblocks, and wherein the motion projection uses the motioninformation associated with macroblocks of the first video frame.